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IMAGE PROCESSING METHOD 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 

The present invention relates to an image processing 
method for converting input image data to output image data 
by applying image processing to the input image data. More 
specifically, the invention relates to an image processing 
method for preparing an output image meeting a request from 
an individual customer; an image processing method for 
additionally displaying information corresponding to a 
feeling of a person on an image displaying medium or the 
like on which an image of the person is displayed, or 
performing substitution, modification or adjustment to 
produce an image corresponding to the feeling; and an image 
processing method for changing an image of a person to a 
favorite image of the person or an image having no 
unnatural feeling . 
2. Description of the Related Art 

Conventionally, printing of an image photographed on 
a photographic film (hereinafter referred to as a film) 
such as a negative film or a reversal film to a 
photosensitive material (printing paper) has been performed 
by so-called direct exposure. With the direct exposure, 
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the image on the film is projected on the photosensitive 
material and the surface of the photosensitive material is 
exposed to light. 

On the other hand, in recent years, a printing 
apparatus that utilizes digital exposure, that is, a 
digital photoprinter, has been put to practical use. The 
digital photoprinter photoelectrically reads an image 
recorded on a film, converts the read image to a digital 
signal, then converts the digital signal to image data for 
recording by applying various kinds of image processing, 
and records an image (latent image) by scanning and 
exposing a photosensitive material by recording light that 
is modulated according to the image data to have a print 
(photograph) - 

The digital photoprinter can convert an image into 
digital image data and determine exposure conditions at the 
time of printing the image by image data processing. Thus, 
various kinds of image processing can be performed with a 
high degree of freedom, which is difficult or impossible 
with the conventional direct exposure, including correction 
of dropouts or blocked-ups of an image due to back-light, 
strobe photographing or the like, correction of a color 
failure or a density failure, correction of under exposure 
or over exposure, correction of insufficient marginal 
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luminosity, sharpness processing, and compression/expansion 
processing of density dynamic range (giving a dodging 
effect by image data processing) . Therefore, an extremely 
high-grade print can be obtained compared with the direct 
exposure. Moreover, since composition and division of a 
plurality of images , composition of characters or the like 
can be performed by the image data processing, a print that 
is freely edited and/or processed depending on an 
application can be outputted. 

In addition, since the digital photoprinter not only 
can output an image as a print but also can supply image 
data to a computer or the like or can store image data in a 
recording medium such as a floppy disk, the image data can 
be utilized for various applications other than a 
photograph. 

In this way, with the digital photoprinter, it is 
possible to apply image processing with a higher degree of 
freedom to an image by the image data processing and to 
output a print with higher commodity value. 
Incidentally, it is preferable that an image to be 
reproduced as a print is an image on which a request of a 
customer (a person who requests preparation of a print) is 
reflected as much as possible, in this regard, the 
applicant has proposed an image processing method of 



reproducing a finished image that preferably corresponds to 
a request of a customer in Japanese Patent Application 
Laid-open No. Hei 11-331570. 

That is, the method is to obtain a reproduced image 
preferably corresponding to a request of a customer by 
getting information on the customer relating to image data 
supplied from an image supplying source,, setting image 
processing conditions according to the information on the 
customer, and performing image processing based on the 
image processing conditions . 

In the laid-open patent application/ information on a 
customer refers to an occupation of a customer a sex of a 
customer, an age of a customer or the like. In addition, a 
method of obtaining information on a customer is 
exemplified by a method with which information on a 
customer is verbally obtained from the customer when an 
order of a print is received from the customer, which is 
communicated to an operator who inputs the information 
using an operating device such as a mouse, a method with 
which customer information is written in a customer card 
and an operator inputs the customer information referring 
to the customer card when preparing a print, or a method 
with which customer information is arranged as a database 
and an operator obtains the customer information from the 
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database* 

In addition, image processing that preferably 
corresponds to a request of a customer is exemplified by 
the following processing. In a case in which a film is a 
reversal film and an occupation is a professional 
photographer,, an image photographed on the film is to be 
reproduced faithfully, and in a case in which an occupation 
is not a professional photographer, a photographing failure 
such as over exposure, under exposure and back-light is 
remedied by adjusting a color and a density of an image 
normally. In addition, in a case of a male, a face region 
is extracted and sharpness is given rather strongly to make 
gradation prominent and show details, and in a case of a 
female, a face region is extracted and sharpness is given 
rather weakly or soft focusing is applied extremely weakly 
to make gradation less prominent (soft) and to make live 
spots, wrinkles, freckles or the like less outstanding. 

However, the conventional image processing method has 
a problem in that processing is complicated because an 
operator must input information on a customer. In addition, 
there is also a problem in that image processing conditions 
to be set are fixed according to obtained information on a 
customer, or selection of conditions is limited only to 
whether the processing is performed or not, and there is no 



function of setting image processing conditions 
corresponding to preference of a customer or, more 
meticulously, of each subject person, thus, image 
reproduction to which preference of a customer or a subject 
person is truly reflected cannot be realized. 

As image forming media, there are conventionally a 
photograph (print) that reproduces a still image and a 
movie (a film, projector and a screen) that reproduces 
images as an animation. Since the development of a 
cathode-ray tube (CRT), in recent years r television sets 
(TVs) have been spread to all the households. Moreover, 
with remarkable advances of technologies r various image 
display devices such as a liquid crystal display, a plasma 
display and an electronic paper have been developed as 
image forming media. 

Recently, image forming devices have been developed 
such as a video camera, a digital camera, a digital video 
movie camera, and a cellular TV telephone, which can 
capture voices together with images utilizing the above-^ 
mentioned image forming media. 

However, although the above-mentioned conventional 
image forming devices can photograph images and, at the 
same time, record voices, the captured voice data is simply 
reproduced as sounds directly. In addition, there is also 
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a problem in that the conventional image forming devices 
aim principally at reproducing an image as faithfully as 
possible as it is photographed, and an entertaining aspect 
of an image is not taken into account at all. 

Further, in the present digital image processing 
technology, it is possible to adjust density or color tone 
of an image photographed and captured as digital image data/ 
and image processing such as correction of the photographed 
image per se and composition or substitution with other 
images is also possible* Various digital image processing 
techniques have been thus proposed which include digital 
image processing technique conventionally performed in the 
field of photograph for correcting or adjusting a face of a 
person or a line of sight. 

For example, Japanese Patent Application Laid-open No. 
2000-151985 discloses a technique in which portions of an 
image of a face of a person and adjustment parameters are 
set to correct the image to have a made-up face. Ordinary 
people are not however familiar with adjustment of color 
tone or gradation, which is difficult processing for 
amateurs and in particular rather difficult for a user 
unfamiliar with a personal computer (hereinafter also 
referred to as "PC"). In some cases, good adjustment 
results cannot be obtained, and there is also a problem in 
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that images very often have unnatural feeling after 
adjustment. 

Further, Japanese Patent Application Laid-open No. 5- 
205030 discloses a technique in which an image of eyes in a 
full-faced state is prepared from a three-dimensional model 
of an image of a face of a person by computer graphics 
(hereinafter referred to as fr CG") technology. There is 
however a problem in that unnatural feeling remains on the 
image of eyes prepared by the CG technology. There is also 
a problem in that arithmetic computations enormously 
increase in quantity because of the preparation from the 
three-dimensional model depending on the CG technology. 

On the other hand, among the digital image processing 
mentioned above, simple image processing can be performed 
with a personal computer, and to this end f various types of 
image processing software programs are commercially 
available. Nevertheless/ the simple image processing by 
using these image processing software programs that are 
commercially available cannot provide a sufficient accuracy 
to finish an image of a face of a person so as to have a 
favorite face or to finish the image in a full-faced state 
by making a line of sight coincident with a photographing 
direction. There is also a problem in that the operation 
is difficult for amateurs. 
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SUMMARY OF THE INVENTION 

The present invention has been devised in view of the 
above drawbacks, and it is a first object of the present 
invention to provide an image processing method with which 
a reproduced image, on which preference of each subject 
person is reflected, can be automatically obtained. 

The present invention has been devised in view of the 
above drawbacks, and it is a second object of the present 
invention to provide an image processing method with which 
amusement aspect in image forming media such as a 
photograph, a video, a TV telephone and the like can be 
enhanced by visualizing a content that is desired to be 
emphasized according to a type of feeling of a person in a 
photographed image/ particularly an image of the person, 
and forming an image . 

The present invention has been devised in view of the 
above drawbacks r and it is a third object of the present 
invention to provide an image processing method with which 
even inexperienced or unskilled persons in personal 
computer or image processing software can easily correct 
images so as to have a preferred made-up face or a favorite 
face and remove unnatural feeling due to noncoincidence of 
the line of sight. 
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In order to attain the first object described above, 
the first aspect of the present invention provides an image 
processing method for applying image processing to an 
inputted image data, comprising the steps of registering 
predetermined image processing conditions for each specific 
person in advance; extracting a person in the inputted 
image data; identifying the extracted person to find if the 
extracted person is the specific person; and selecting 
image processing conditions corresponding to the identified 
specific person to perform the image processing based on 
the selected image processing conditions. 

Preferably, the extracted person is identified using 
a face image of the specific person registered in advance 
or person designation information accompanying a 
photographed frame. 

Preferably,, a plurality of kinds of image processing 
conditions are set for the each specific person as the 
predetermined image processing conditions to be registered 
for the each specific person in advance. 

Preferably, the image processing is performed by 
using at least one image processing condition selected from 
the plurality of kinds of image processing conditions. 

Preferably, it is set whether the image processing 
under the selected image processing conditions is applied 
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to an image as a whole or applied only to the person or the 
person and a vicinity of the person. 

In order to attain the second object described above, 
the second aspect of the present invention provides an 
image processing method, comprising the steps of 
determining a type of feeling from types of feeling 
registered in advance based on at least one kind of 
information selected from among voice data accompanying a 
photographed image, an expression of a person extracted 
from the photographed image, and a gesture of the extracted 
person; and subjecting the photographed image to image 
processing which applies an image processing pattern 
corresponding to the determined type of feeling among image 
processing patterns set in advance. 

Preferably, the image processing pattern is set in 
association with the type of feeling, and the image 
processing to which the image processing pattern is applied 
is at least one processing selected from among composition 
processing for composing a specified mark corresponding to 
the type of feeling, substitution processing for 
substituting with an animation image or a computer graphics 
image corresponding to the type of feeling, image 
modification processing performed on the photographed image 
in correspondence with the type of feeling, and processing 
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for changing a density and a color of the photographed 
image in correspondence with the type of feeling. 

Preferably,, the composition processing is processing 
for composing the specified mark at a predetermined 
position in the photographed image or at a predetermined or 
relative position with respect to the person extracted in 
advance or during the composition processing from the 
photographed image, and in a predetermined or relative size 
and a predetermined or relative orientation with respect to 
the photographed image or the extracted person. 

Preferably, the substitution processing is processing 
for substituting a specified portion of the person 
extracted in advance or during the substitution processing 
from the photographed image with the animation image or the 
computer graphics image. 

Preferably, the photographed image is a photographed 
image by an image photographing device with a recording 
function, and the image processing pattern is registered in 
the image photographing device with the recording function 
in advance, and the image processing to which the image 
processing pattern is applied is performed by the image 
photographing device with the recording function. 

Preferably/ the image processing to which the image 
processing pattern is applied is performed on a lab side 
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that receives image photographing information including the 
voice data recorded by the image photographing device with 
the recording function. 

Preferably, the photographed image is a photographed 
image by a telephone call device with a photographing 
function, and the image processing to which the image 
processing pattern corresponding to the type of feeling of 
the person is applied is performed on the photographed 
image - 

Preferably, the image processing pattern is 
registered in the telephone call device with the 
photographing function in advance, and the image processing 
is performed by the telephone call device with the 
photographing function to transmit a processed image to a 
terminal on an opposite party side. 

Preferably, the image processing pattern is 
registered in a repeater station of the telephone call 
device with the photographing function in advance, and the 
image processing is performed in the repeater station to 
transmit a processed image to one terminal in a connected 
telephone line. 

Preferably, the image processing pattern is 
registered in the telephone call device with the 
photographing function in advance, and the image processing 
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to which the image processing pattern is applied is 
performed by the telephone call device with the 
photographing function on an image that was photographed by 
a terminal on an opposite party side and received by the 
telephone call device with the photographing function. 

Preferably, if a specified mark corresponding to the 
type of feeling or a composing position of the mark is 
wrong in the image processing to which the image processing 
pattern is applied, the mark corresponding to the type of 
feeling, the composing position of the mark and a size or 
an orientation of the mark can be corrected* 

In order to attain the second object described above, 
the third aspect of the present invention provides an image 
processing method comprising the steps of capturing a 
television image in a personal computer; and performing 
image processing to which an image processing pattern is 
set in advance on the captured television image in the 
personal computer. 

In order to attain the third object described above, 
the fourth aspect of the present invention provides an 
image processing method comprising the steps of registering 
in advance an area image in a specific area of an image or 
an image characteristic amount; and composing on a 
corresponding area of an photographed image or adjusting a 
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density and a color tone by using the area image or image 
characteristic amount registered in advance. 

Preferably, the corresponding area is extracted from 
the photographed image in accordance with the area image or 
the image characteristic amount registered in advance. 

Preferably, the specific area is at least one of a 
face of a person, at least one portion constituting the 
face of the person, an accessory that the person wears and 
a background* 

Preferably/ the specific area is a face of a person, 
and the area image registered in advance is an image of a 
made-up face or best face of the person. 

Preferably, the specific area is an area of eyes 
constituting a face of a person, and determination is made 
as to whether the person as a subject in the ■ photographed 
image is in a stationary state, and when the person is in 
the stationary state, the area image in the specific area 
registered in advance is composed on the area of eyes 
constituting the face of the person. 

Preferably, the area image registered in advance is 
an image of the area of eyes in which a line of sight of 
the person is coincident with a photographing direction* 



BRIEF DESCRIPTION OF THE DRAWINGS 
In the accompanying drawings; 

Fig. 1 is a block diagram schematically showing an 
example of a digital photoprinter including an image 
processing apparatus that implements an image processing 
method in accordance with a first aspect of the present 
invent ion ; 

Fig* 2 is an explanatory illustration showing an 
example of registration data used in the present invention; 

Fig, 3 is a flow chart showing operations of a first 
embodiment of the first aspect of the present invention; 

Fig. 4 is a block diagram schematically showing an 
example of a digital photoprinter including an image 
processing apparatus implementing an image processing 
method in accordance with a second aspect of the present 
invention; 

Fig. 5 is a flow chart showing a flow of processing 
of an image processing method in accordance with a first 
embodiment of the second aspect of the present invention; 

Fig, 6 is an explanatory illustration showing an 
example of an image processing pattern; 

Fig. 7 is an explanatory illustration schematically 
showing a cellular TV telephone system in accordance with a 
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fourth embodiment of the second aspect of the present 
invention; 

Fig* 8 is a block diagram schematically showing an 
example of a digital photoprinter including an image 
processing apparatus implementing an image processing 
method in accordance with a fourth aspect of the present 
invention; and 

Fig. 9 is a flow chart showing a flow of processing 
of image processing methods in accordance with first and 
second embodiments of the fourth aspect of the present 
invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

An image processing method in accordance with the 
present invention will be hereinafter described in detail 
based on preferred embodiments of the present invention 
shown in the accompanying drawings. 

In the following description of the embodiments, 
reference is made to drawing figures. Like reference 
numerals used throughout the several figures refer to like 
or corresponding parts. 

An image processing method in accordance with a first 
aspect of the present invention will be described first 
with reference to Figs. 1 to 3. 
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Fig. 1 is a block diagram schematically showing an 
example of a digital photoprinter including an image 
processing apparatus that implements an image processing 
method of automatically obtaining a reproduced image, on 
which preference of an individual customer is reflected, in 
accordance with a first aspect of the present invention, 

A digital photoprinter (hereinafter referred to as a 
photoprinter) 10 shown in Fig. 1 includes a scanner (image 
reading apparatus) 12 for photoelectrically reading an 
image photographed on a film F, an image processing 
apparatus 14 for performing image processing such as 
electronic magnification processing of image data read by 
the scanner 12, edge detection and sharpness emphasis of 
image data and smoothing processing (granularity 
restraining) f or operation, control and the like of the 
entire photoprinter 10, and an image recording apparatus 16 
for applying image exposure and developing processing to a 
photosensitive material {printing paper) by light-beam that 
is modulated according to the image data outputted from the 
image processing apparatus 14 to output a (finished) image 
as a print- 

In addition, an operation system 18 including a 
keyboard 18a and a mouse 18b for inputting a selection or 
an instruction of input, setting and processing of various 
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conditions, an instruction of color /density correction, or 
the like, and a monitor 20 for displaying the image read by 
the scanner 12, various operational instructions, a 
setting/registration screen of various conditions, and the 
like are connected to the image processing apparatus 14. 

The scanner 12 is an apparatus for photoelectrically 
reading an image photographed on a film F or the like one 
frame after another, which includes a light source 22, a 
variable diaphragm 24, a diffusion box 26 for equalising 
reading light incident on the film F in the surface 
direction of the film F, a carrier 28 of the film F, an 
imaging lens unit 30, an image sensor 32 having a three- 
line CCD sensor dealing with reading of each color image 
density of R (red) , G (green) and B {blue) , an amplifier 33 
and an A/D (analog/digital) converter 34. 

In the photoprinter 10, the carrier 28 that is for 
special -use with and detachably attachable to the body of 
the scanner 12 is prepared according to a type and a size 
of the film F such as a film of the Advanced Photo System, 
a negative {or reversal) film of the 135 size or the like, 
a form of a film such as a strips and a slide, or the like. 
Thus, the photoprinter 10 can cope with various kinds of 
films and processing by replacing the carrier 28. An image 
(frame) that is photographed on a film and is served for 
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preparing a print is conveyed to a predetermined reading 
position by the carrier 28. 

In addition,, as is well-known, a magnetic recording 
medium is formed in a film of the Advanced Photo System, on 
which an cartridge ID and a film size, an ISO sensitivity, 
and the like are recorded. Further, various data such as a 
date and time of photographing or development, an exposure 
level, a type of a camera or a developing machine can be 
recorded at the time of photographing, development or the 
like. A reader of this magnetic information is disposed on 
the carrier 28 dealing with a film (cartridge) of the 
Advanced Photo System, which reads the magnetic information 
when the film is conveyed to a reading position. The read 
various information is sent to the image processing 
apparatus 14. 

In the scanner 12 as described above, when an image 
photographed on the film F is read, uniform reading light, 
which is irradiated from the light source 22 and whose 
amount is adjusted by the variable diaphragm 24 and the 
diffusion box 26, is incident on and transmits through the 
film F located in the predetermined reading position by the 
carrier 28. A projected light bearing the image 
photographed on the film F is thereby obtained. 

Further, a color image signal is not limited to a 
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signal inputted by reading light that has transmitted 
through a film, but a reflected original or an image 
photographed by a digital camera may be used. That is, an 
image (digital image signal) can be inputted in the image 
processing apparatus 14 from, other than the scanner 12 for 
reading an image of a film, an image supplying source R, 
such as image photographing devices such as a digital 
camera and a digital video camera, an image reading 
apparatus for reading an image of a reflected original, 
communication networks such as an LAN (Local Area Network) 
and a computer communication network, and a medium 
(recording medium) such as a memory card, such as a smart 
medium, and an MO (mangetooptical recording medium) ♦ 

The illustrated carrier 28 deals with an elongated 
film F (strips) such as a 24-exposure film of the 135 size 
and a cartridge of the Advance Photo System. 

The film F is placed in a reading position by the 
carrier 28, on which reading light is irradiated while it 
is conveyed in a sub-scanning direction perpendicular to a 
main scanning direction that is an extending direction of 
the three-line CCD sensor of R, G and B. Thus, as a result, 
the film F is slit-scanned two-dimensionally and an image 
of each frame photographed on the film F is read. 

Projected light of the film F is imaged on a light- 
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receiving surface of the image sensor 32 by the imaging 
lens unit 30. 

Each output signal of R, G and B outputted from the 
image sensor 32 is amplified by the amplifier 33 and sent 
to the A/D converter 34. In the A/D converter 34, each 
output signal is converted to R, G and B digital image data 
of,, for example, 12 bits, and then outputted to the image 
processing apparatus 14. 

Further, in the scanner 12, image reading is 
performed twice, i.e., pre-scan for reading an image with 
low resolution (first image reading) and fine scan for 
obtaining image data of an output image (second image 
reading) , when an image photographed on the film F is read. 

Here, the pre-scan is performed under pre-scan 
reading conditions set in advance such that the scanner 12 
can read all the images on the film F, which are objects of 
the scanner 12, while the image sensor 32 does not saturate. 

On the other hand, the fine scan is performed under 
reading conditions of the fine scan set for each frame such 
that the image sensor 32 saturates at a density slightly 
lower than a lowest density of the image (frame) . Further, 
output image signals of the pre-scan or the fine scan are 
basically similar image data except that resolutions and 
output image signal levels are different. 
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Further, the scanner 12 used in the photoprinter 10 
is not limited to a scanner that performs such slit-scan 
reading, but may be a scanner that performs plane reading 
for reading the entire surface of a film image of one frame 
at a time. 

In this case, the scanner 12 uses an area sensor such 
as an area CCD sensor, and an inserting device for each of 
the color filters of R, G and B is provided between the 
light source 22 and the film F. The color filter is 
inserted in an optical path of light irradiated from the 
light source 22. Reading light that has transmitted 
through the color filters is irradiated on the entire 
surface of the film F to cause the transmitting light to 
focus on the area CCD sensor for reading the entire image 
of the film. The scanner 12 dissolves an image 
photographed on the film F into three primary colors to 
read them by replacing each color filter of R, G and B one 
after another to repeat this processing. 

Rs described before, the digital image data signal 
outputted from the scanner 12 is outputted to the image 
processing apparatus 14 that implements the image 
processing method in accordance with this aspect of the 
present invention . 

In order to first correct dispersion of sensitivities 
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for each pixel of the R, G and B digital image data and 
dark current due to the CCD sensor with respect to the 
image signals of R, G and B inputted in the image 
processing apparatus 14 from the scanner 12, data 
correction of read image data such as DC offset correction, 
dark state correction, defective image correction and 
shading correction is performed. Thereafter, the image 
data is subject to logarithmical conversion processing and 
gradation conversion, and is converted into digital image 
density data. 

The pre-scan image data and the fine scan image data 
in the digital image density data are separately memorized 
(stored) . The pre-scan image data is subject to a 
predetermined image processing and displayed on the monitor 
20. In addition, a density histogram is prepared and image 
characteristic volumes such as an average density, an LATD 
(large area transmission density) , a highlight (minimum 
density) and a shadow (maximum density) are calculated from 
the pre-scan image data to set reading conditions and image 
processing conditions of the fine scan. In this aspect of 
the present invention, image processing conditions are set 
such that individual preference of each customer is 
reflected on the conditions as described later. Image 
processing is applied to the fine scan image data according 
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to the set image processing conditions, and is outputted on 
color paper from the image recording apparatus 16 as an 
optimal, high quality image at a density, gradation and 
tone that are desired as a color print. 

The digital photoprinter including the image 
processing apparatus that implements the image processing 
method in accordance with the first aspect of the present 
invention is basically configured as described above. 

Operations of a first embodiment of the first aspect 
of the present invention will be hereinafter described. 

As described above, the image processing according to 
individual preference of each customer is executed in the 
image processing apparatus 14 together with the processing 
such as conversion of a density, a color and gradation of 
image data, conversion of a chroma and electronic 
magnification with respect to the fine scan image data that 
is read- with the image reading conditions that are set 
based on the pre~scan image data, 

The image processing on which individual preference 
of each customer is reflected will be hereinafter described. 

First, as a precondition,- data such as individual 
identification information and image processing conditions 
desired by each individual (request processing information) 
is registered in a lab in advance in order to identify a 



person in an image and execute image processing according 
to preference of the specific person. 

An example of registration data is shown in Fig. 2, 
The example shown in Fig. 2 is in a case in which data of 
all members of a family is registered. 

Contents of the registered data are, for example, 
registered customer IDs, a name of a person representing 
the people to be registered and individual data of each 
member of the family, i.e., a relation in the family/ a 
face image of each individual, image processing contents 
desired by each individual (request processing) 7 and the 
like. 

Face image data may be captured by designating, for 
example, a face area, in an already ordered photographed 
scene. In addition, a plurality of patterns for each 
person may be set and registered in order to increase 
accuracy of identification* 

In addition, as examples of request processing, there 
are adjustment of a density/color tint of a face, 
conversion of a color of hair, eyes, rouge or the like to a 
designated color, white finish of teeth, a special effect, 
slim body finish, image composition and the like. 

Here, as adjustment of a density/color tint of a face, 
a mode for changing skin to white, a mode for erasing 
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wrinkles, a suntan mode and the like can be considered. As 
a special effect, there are use of a cloth filter, soft 
focus finish and the like. As image composition, there are 
frame decoration, composition of specific characters, 
special make-up and the like* 

A plurality of kinds of the above-mentioned request 
processing may be set for each person, and some selected 
kinds of request processing out of the plurality of kinds 
of the request processing may be combined to be applied to 
image processing of a specific person. 

Moreover, depending on processing contents, 
conditions such as whether the processing is applied to the 
entire image, or applied only to a pertinent person or only 
to the person and the vicinity of the person are set. 

Image processing on which individual preference of 
each customer is reflected will be hereinafter described 
with reference to a flow chart of Fig. 3. 

In step 110, a face region of a person is extracted. 
An extracting method of a face region is not specifically 
limited but there are a variety of methods. It is 
sufficient if a publicly known specific part extracting 
method (extraction algorithm) is used* 

Such a specific part extracting method is exemplified 
by a method with which a plurality of different specific 
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part (major part) extracting methods, such as a method of 
extracting a specific color, a method of extracting a 
specific shape pattern, and a method of removing a region 
that is estimated to be equivalent to a background 
disclosed in Japanese Patent Application Laid-open No, Hei 
9-138470, are evaluated to define weights in advance, a 
specific part is extracted by each extracting device, the 
extracted specific part is weighted by the defined weight, 
and the major part is determined and extracted according to 
a result of the weighting* 

In addition, another specific part extracting method 
is exemplified by a method with which densities or 
luminance of a plurality of points in an image are measured 
to find variations among the points, a point with a 
variation equal to or more than a predetermined value is 
set as a standard point, then a retrieval area and a 
retrieval direction pattern are set using a variation or 
the like of a density or the like within a predetermined 
range from the standard point to retrieve a part with a 
variation of a density or the like in the direction 
indicated by the retrieval direction pattern within the 
retrieval area, and subsequently the retrieval is repeated 
with this part as a standard point to extract a specific 
part by connecting retrieved/set standard points as 



disclosed in Japanese Patent Application Laid-open No. Hei 
9-138471. 

In this aspect of the present invention, specific 
part extracting methods disclosed in each o£ Japanese 
Patent Application Laid-open No* Hei 4-346333, 5-158164, 5- 
165120, 6-160993, 8-122944, 8-184925, 9-101579, 9-146194, 
9-19757 5 and the like are preferably available other than 
the above-mentioned methods. 

When a face is extracted, a person is identified and 
a person in the image is specified in step 120. 

After normalizing sizes, matching is performed with 
respect to the extracted face image with registered face 
images as a template to find a coinciding degree, and a 
person is identified. At this point, if information 
indicating the person in the image exists as information 
relating to a comment accompanying the image, the 
information may be utilized for identification of a person* 

If a person is successfully specified as a result of 
the identification of a person, request processing for the 
registered person is executed with respect to the specific 
person in step 130. 

In this way, according to this embodiment, a finished 
image according to individual preference of each customer 
can be automatically obtained* For example, an image of 
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the father is processed to be a suntanned face, soft focus 
and wrinkle erasing processing are applied to an image of 
the mother, and white skin and slim body finish is applied 
to an image of a daughter A* Thus, finished images as the 
members of the family wish to have are obtained, 
respectively. 

h second embodiment of the first aspect of the 
present invention will now be described. 

The second embodiment is to execute processing as 
described in the first embodiment in a case in which a 
customer performs reproduction processing of an image using 
a digital camera, a personal computer, a photo player or 
the like* That is, before the customer places an order 
with a lab, a photographed image is displayed on these 
devices to revise the image. Causing a digital camera or a 
photo player to have a revising function, or connecting the 
devices with a personal computer can revise an image. 

In addition, if a displayed image is revised, 
although the revision can be automatically performed using 
image processing software for performing processing similar 
to that of the first embodiment, a customer may manually 
identify a person and revise a result of detecting a person 
while looking at the displayed image, in addition, an 
image of a result of the revision may be printed using a 
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video printer for home use, or revision data of a frame may 
be recorded as information attached to the frame to order a 
print for the lab. For example, it is sufficient to 
display an image photographed by the APS on a photo player 
to revise the image, and if an order is placed with the lab 
to print from a negative film, to cause the photo player to 
have a magnetic recording function and to magnetically 
record the attached information in the negative film. In 
addition, if an order is placed with the lab using a 
communication line by a personal computer, it is sufficient 
to record the attached information in a file. 

A third embodiment of the first aspect of the present 
invention will now be described. 

As described in the first embodiment, this embodiment 
is to improve accuracy of identification of a person by 
adding a hair style, a dress and the like as person 
designation information to characteristic amounts for 
recognition of a person, when a face is extracted and 
template matching is performed between the face and the 
registered face image in identification of a person. Here, 
as the characteristic amounts for recognition of a person, 
a density, a distribution of color tint, a texture pattern 
or the like can be considered. 

In this embodiment, as in the first embodiment, at 
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first a face region is extracted from an input image to 
perform matching with registered face images as a template, 
to find a coinciding degree and to identify a person. Then, 
if a person in an image is specified by the identification 
of a person, a hairstyle, a dress and the like of the 
person are extracted, which are registered as person 
designation information and as registration data 
corresponding to the person. 

In the next processing on the input image , 
identification of a person is performed with reference to 
the hairstyle, the dress and the like that have been 
registered before in addition to by means of matching of 
the template of the extracted face image. When a person is 
specified by the identification of a person, a hairstyle 
and a dress are registered anew from the input image to 
automatically accumulate them as person designation 
information. 

For example, it is assumed that data accumulated to 
the present concerning color tint of dress areas is as 
shown in Table 1. This Table 1 classifies color tint of 
dress areas and indicates frequencies of colors appearing 
in each member of a family by pi j - 
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Table 1 



i: Person 


Classifications of color tint 




Red 


Orange 


Yellow 






Achromatic 
color 


1. Father 


0 






Plj 




plJ 


2. Mother 


3 






P2j 




p2J 


3. 

Daughter A 


10 






P3j 




p3J 



J indicates the number of classifications of color tine. 
If a dress is composed of a plurality of colors r an area 
ratio is used for classifications. For example, if a half 
of a dress is red and the other half is white, red=0.5 and 
white-0.5 are registered in accumulated data- 

Here, it is assumed that Mi is a coinciding degree by 
the pattern matching of a face image represented by a point, 
and Ni is a point derived from a frequency of appearance of 
a dress. The large each coinciding degree, the large a 
point is. In Table 1, i is a number indicating a 
registered person. For example, the daughter & is i=3- It 
is assumed that, if color tint of a dress area of a subject 
person being an object of photographing corresponds to a 
classification number j , Ni = pij /(plj + p2j + p3 j ) • 

By finding a person determination point Di - a x Mi + 
b x Ni (a and b are weighting factors) , it is determined 
that a person with the number i being the largest point is 
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the person in the image. 

For example, it is assumed that Ml < M2 == M3 as a 
result of the matching of a face image,, thus the mother and 
the daughter A have substantially the same points- Here, 
if the color tint of a dress is red, since the number of 
times each person is photographed while dressed in red is 0 
for the father, 3 for the mother and 10 for the daughter A, 
the daughter A has the largest point Di. Therefore, the 
person in the image is determined to be the daughter A. 

If there is an error in the identification of a 
person, an operator who has found the error or an operator 
who has been notified by a customer makes a revision 
manually, performs processing with respect to a correct 
person in the image and, at the same time, corrects 
accumulation processing of the person designation 
information. 

In addition, parameters for a determination algorithm 
for identification of a person are automatically revised at 
this point _ 

In addition, a customer who has found an error in 
identification of a person may revise the processing by 
himself if the customer revises an image in person by 
downloading image processing software in the customer's own 
personal computer as in the second embodiment. 
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According to this embodiment, in addition to simply 
performing the template matching of an image with a 
registered face image, person designation information such 
as a hairstyle and a dress is also added to characteristic 
amounts for recognition of an individual/ and moreover, 
accumulated data is revised by adding an element of 
learning function to the accumulation processing of the 
person designation information* Therefore, accuracy of the 
identification of a person can be improved, and image 
processing according to preference of each person can be 
properly per formed . 

The above description relates to a method of 
obtaining a print (output image) on which individual 
preference of each customer is reflected by applying image 
processing according to the individual preference of each 
customer to each person in an image. However, in some 
cases, it is desired that a print is outputted on which the 
tendency of preference according to locality of a customer 
or a season is reflected in addition to matching individual 
preference of each customer. 

For example, in a case in which an exposure control 
algorithm is set such that a photo print is finished in a 
skin color preferred by the Japanese in a photoprinter, if 
a printer of the same type is used in Europe without 
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changing the exposure control algorithm, a skin color may 
not be a color preferred by the Europeans. In addition, in 
a case in which the exposure control algorithm is set based 
on photographing under the sunlight in Japan, if a photo 
printer of the same type is used in a region at a 
substantially different latitude, a print of the same 
quality as that in Japan cannot be realized because the 
sunlight is also different in this region from that in 
Japan . 

Thus, the applicant has already proposed an exposure 
control method with which local preference of a person or 
preference of an individual customer is properly reflected 
on a print in the case of the analog exposure method in 
Japanese Patent Application Laid-open No. Hei 6-308 632 or 
Japanese Patent Application Laid-open No, Hei 8-137033. 

A print (output image) on which locality or the 
tendency of preference of a customer is reflected can also 
be realized in the case of the digital image processing as 
described below. 

That is, an example described below is a digital 
image processing method of accumulating the tendency of 
revising a setting of image processing conditions for each 
customer, lab or region to revise and optimize a setting 
parameter of image processing by* In addition, the method 
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is also for classifying each frame by scene to accumulate 
the tendency of revision for each scene classification, and 
revise and optimize the setting parameter. 

For example, when performing image processing, if a 
correction tendency of an operator shows, after classifying 
each frame by scene, varying tendencies for a certain 
region such as that it is preferred to emphasize a contrast 
or to be moderate in a contrast, the operator revises a 
correction parameter in accordance with the tendencies. 

Describing the processing method more specifically, 
revisions of a processing parameter by an operator are 
accumulated for the number of N frames in the past to check 
relation with a specific scene classification. At this 
point, there are the following methods as methods of 
classifying scenes . 

For example, a scene is estimated as a strobe scene, 
a back-light scene or the like by estimating a normal scene, 
over exposure, under exposure or the like based on a 
distribution pattern found from a density histogram or the 
relation between density differences between a central part 
and a peripheral part of an image. 

Alternatively, a scene is estimated to be a person, a 
scenery or the like by extracting a face. There are some 
other methods such as a method with which an operator 



38 

manually estimates a classification of a scene such as a 
person, a scenery; a night scene, underexposure or high 
contrast. 

In addition, as an example of changing a processing 
parameter, there is such a method with which a way of 
applying contrast emphasis or resolution (sharpness) is 
changed or a target value of color tint or a density of a 
skin color of a person is changed. Since a person and a 
preferable skin color varies depending on a region/ this 
method takes into account the difference due to a region. 
Since, in Europe, a color of a face that is less dense than 
a skin color based on the Oriental people is preferred and 
an operator tends to reduce a density, an algorithm is 
automatically updated such that a density becomes slightly 
lower than a default value after a face is extracted 
judging from accumulated data. 

As -described above, changes of a processing parameter 
by an operator are classified for the number of N frames in 
the past, relation between the classification and a 
specific scene classification is found, Then, an algorithm 
is updated and an image processing parameter is optimized 
in a fixed cycle by reflecting a statistic manual 
correction tendency for each image processing parameter in 
a predetermined scene classification. 
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Further, in this case, it is preferable to also 
update accumulated data to latest one in a fixed cycle . 

In addition, in the above-mentioned processing, types 
of scene classifications or a data accumulation period may 
be set for each season. For example, around a ski resort, 
a fT snow scene" classification that is determined according 
to a highlight ratio in a scene may be added only in the 
winter season, or a gradation characteristic emphasizing 
whiteness may be applied to the "snow scene". 

In addition, in a fixed region, algorithm updating 
processing of the region may be performed by a unit of 
region in a lab management center. This can be attained by 
connecting each mini-lab shop in the region and the lab 
management center in the region by a network such that the 
lab management center collects information on revisions by 
an operator in each mini-lab shop, grasps a tendency of the 
entire region, revises image processing software of mini- 
lab shops all over the region at a certain timing, and 
distributes an updated algorithm to each mini-lab shop. 

In addition, when performing the above-mentioned 
processing, an image processing parameter may be optimized 
for each customer by recording the data together with 
customer IDs and managing the data for each customer. 

Further, the above-mentioned processing can be 
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performed with respect to a printer for home use of each 
customer instead of a printer in the above-mentioned mini- 
lab. 

According to the example mentioned above, an 
appropriate image reproduction becomes possible which copes 
with the change of a tendency of preference for each scene 
due to an individual customer, locality, a season or the 
like. 

In addition, when a photoprinter is manufactured, it 
becomes unnecessary to set every parameter to conform to a 
tendency of each region being a destination of the product 
and operations for manufacturing and delivering the product 
is facilitated. 

In addition, request contents may be divided for each 
of print processing and monitor display processing, or 
registration data processing can be stopped or specific 
request . processing can be designated only for a specific 
frame. 

The image processing method of the first aspect of 
the present invention is basically configured as described 
above . 

Image processing methods in accordance with second 
and third aspects of the present invention will now be 
described with reference to Figs. 4 to 7 . 
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The image processing method in accordance with the 
second aspect of the present invention is basically to 
determine a type of feeling of a person in a photographed 
image scene based on voice data accompanying the image 
scene or based on an expression or gesture of the person 
extracted from the photographed image, and attach (compose) 
a mark emphasizing the feeling corresponding to the type of 
feeling to an image of the person in the scene, thereby 
enhancing amusement aspect of a photograph or an image 
expression, h case in which a type of feeling of a person 
in a photographed image scene is determined based on voice 
data accompanying the photographed image scene will be 
described below as a typical example, but the second aspect 
is not limited to this case. Further, objects to which the 
second aspect of the present invention is applied range 
widely from a photograph (still image) to real time image 
display- such as a video (animation) and a TV telephone, or 
the like. 

A first embodiment of the second aspect of the 
present invention will be described first. The first 
embodiment is for applying predetermined image processing 
with respect to an image having voice data as accompanying 
information which is photographed by a digital camera or 
the like having a voice recording function. 
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Fig- 4 is a block diagram schematically showing a 
digital photoprinter including an image processing 
apparatus that implements the image processing method in 
accordance with the first embodiment of the second aspect 
of the present invention* 

A digital photoprinter 50 shown in Fig. 4 mainly 
includes a photographing information inputting device 52, 
an image processing apparatus 54 and an image recording 
apparatus 16. Further, the image recording apparatus 16, 
the operation system 18 and the monitor 20 similar to those 
in the photoprinter 10 shown in Fig. 1 can be used. 

The photographing information inputting device 52 is 
to read image data and voice data from a recording medium 
in which the image data and the voice data are recorded by 
an image photographing device with a recording function 
such as a digital camera. The image processing apparatus 
54 is to execute the image processing method in accordance 
with this aspect of the present invention and other various 
kinds of image processing. In addition, the operation 
system 18 including the keyboard 18a and the mouse 18b for 
inputting and setting conditions for various kinds of image 
processing, selecting and instructing a particular 
processing, and instructing color/density correction, or 
the like,, and the monitor 20 for displaying the image 
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inputted from the photographing information inputting 
device 52, a setting/registration screen for various 
conditions including various operational instructions, and 
the like are connected to the image processing apparatus 54 
The image recording apparatus 16 is to apply image exposure 
and developing processing to a photosensitive material 
(printing paper) by light-beam that is modulated in 
accordance with rhe image data outputted from the image 
processing apparatus 54 to output a (finished) image as a 
print . 

Image processing of the first embodiment will be 
hereinafter described with reference to a flow chart of Fig 
5. 

First, in step 200, image processing patterns 
corresponding to various types of feelings are set in 
advance* However f since feelings of a human varies and it 
is impossible to deal with all the feelings, only typical 
feelings that relatively clearly appear outside are dealt 
with here. 

Fig. 6 shows an example of setting image processing 
patterns. As shown in the column indicated as a mode, the 
types of feelings to be dealt with here include "fret", 
"surprise", "anger", "sorrow", "worry (doubt)", "love", 
"pleasure" and the like. 



In this embodiment , a type of feeling is determined 
by voice data (voice information) accompanying a 
photographed image scene. In addition, as an image 
processing pattern,, a mark (image composition pattern) is 
composed which emphasizes a feeling with respect to each 
type of feeling. Therefore, setting an image processing 
pattern corresponding to a type of feeling is eventually 
setting an image composition pattern corresponding to 
specific voice information. 

That is, items that are set in advance include a mode 
for representing a type of feeling, a word for drawing each 
mode (keyword, omitted in the table in Fig. 6), and an 
image composition pattern corresponding to each mode. The 
image composition pattern includes a composition image 
(mark) to be composed, a position where an image should be 
composed (a position in an image, a specified or relative 
position with respect to a position of a person or his or 
her face in an image) , a size of a composition image (a 
size, a size with respect to an image, a relative size with 
respect to a size of a person or his or her face in an 
image) , an orientation of a composition image with respect 
to a person or his or her face and the like. 

For example, if a keyword such as "ow", "chickie" or 
"zex" is in voice information, the "fret" mode corresponds 
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to the feeling, and a "sweat mark" is composed with respect 
to an image of a person. A composition image is not 
limited to one, but may be plural. In addition, a 
composing position is preferably designated by a position 
relative to a position of a face in an image and is more 
preferably normalized with a width of the face, h size of 
a composition image is also preferably designated based on 
a face* In addition, an orientation of composing a 
composition image is also preferably set based on a face of 
a person in an image . 

For example, in a case of Fig. 6, two sweat marks 
(one set consists of two marks) are composed in the left of 
a face with a coordinate of a center of the sweat marks in 
the position beside an eye (xl, yl) (with an eye as the 
origin) and in the size of 0.1 when the width of the face 
is 1 * 

In addition, in a case in which a keyword such as 
"whoof" , "jiminy" or "amazing" is in the voice information, 
the rT surprise" mode corresponds to the feeling, and "marks 
of protruding eyes due to surprise" are composed at the 
position where coordinates of the centers of the eye mark 
are (x2, y2) and (x3, y3), respectively. In addition, an 
"orientation determination result" in Fig. 6 indicates that 
an orientation of a person with respect to a camera is 
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determined to be either in front, at the angle to the right 
or at the angle to the left by a pattern matching method or 
the like, and patterns with different orientations of 
protruding eyes depending on the determination result are 
composed. 

In addition, in a case in which a keyword such as 
"cold" or "chilly" is in the voice information, the 
"sorrow" mode corresponds to the feeling, and a "shading 
(slanted lines) mark" is placed on a face and a "falling 
leaves mark" is composed around the face* In such a case, 
the color of the face may be white (pale) instead of 
placing slanted lines in the face* Moreover, more touch of 
sorrow may be added by changing the background to be 
monotone. In addition, other marks may be composed as 
follows. If there is a word such as "Mmm" or "well", a mode 
is determined to be the "worry (doubt) " mode and a 
"question mark" is composed. If there is a word such as 
"aah" or "love", a mode is determined to be the "love" mode 
and a "heart" is composed. If there is a word such as 
"yahoo" or "made it", a mode is determined to be the 
"pleasure" mode and a "fireworks" mark is composed. A 
certain number of these image processing patterns may be 
prepared on the system side and a customer may prepare and 
add data to the patterns. In this case, a word frequently 
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used by the customer in a favorite phrase may be set as a 
word drawing a mode, or a composition image or a keyword 
for various feeling mods may be registered as the customer 
likes . 

It is sufficient to set the above-mentioned image 
processing patterns once prior to respective image 
processing. The setting is performed with respect to a 
database on the lab side* 

Then, in step 210, photographing information having 
voice data as accompanying information which is 
photographed by a digital camera or the like having a 
recording function is inputted form the photographing 
information inputting device 52. Voice data and image data 
in the inputted photographing information are sent to the 
image processing apparatus 54, respectively. 

Then, in step 220, a type of feeling is specified 
from the voice data in the image processing apparatus 54. 
For this purpose, the voice data is recognized first and 
matching is performed to find if a keyword that draws a 
registered mode indicating a type of feeling is included in 
the voice data. If a specific keyword is detected from the 
voice data, a specific image processing pattern is 
specified by a mode corresponding to the keyword. 

In the next step 230, a composition image of the 
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image processing pattern specified above is composed on the 
photographed image- 

In a case in which a composing position of the 
composition image to be composed on the photographed image 
is predetermined with respect to the photographed image in 
a specified position such as any of four corners including 
right upper corner, and in which a specified size or 
relative size of the composition image is predetermined 
with respect to a size of the photographed image,, the 
composition image is composed on the photographed image in 
the predetermined composing position and in the 
predetermined size. 

When the composing position and the composing size of 
the composition image to be composed on the photographed 
image is predetermined with respect to a person or his or 
her face in the photographed image in a fixed or relative 
manner, - the person or the face is preferably extracted from 
the photographed image by a publicly known method in 
advance. As a method of extracting a face, there is, for 
example, a method disclosed in Japanese Patent Application 
Laid-open No. Hei 8^122944 by the applicant of this 
application. Further, as a device for extracting a 
specific region such as a face region of a person, a method 
disclosed in each of Japanese Patent Application Laid-open 
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No. Hei 4-346333, 5-158164, 5-165120, 6-160993, 8-184925, 
9-101579, 9-138470, 9-138471, 9-146194 and 9-197575 is 
preferably available in addition to the above-mentioned 
methods * 

When a face of a person is extracted form the 
photographed image, eyes are extracted from the face, and 
the width of the face, the positions of the eyes and the 
like are calculated. Then, based on the data, a 
composition image is composed in accordance with a 
composing position, a composing size and an orientation 
designated in the image processing pattern. 

In addition, if there are other designations such as 
to change a density or a color of the background or the 
entire screen, processing of such a designation is 
performed* Thereafter, normal image processing is 
performed and an output image is prepared. 

Determination of a type of feeling of a person (face) 
in the photographed image is not limited to the one 
depending on voice data as described above, but the type of 
feeling may be determined by using an expression or a 
gesture of the person, which will be described later in 
detail. 

Further, in determining a type of feeling only from 
voice data as described above, it is unnecessary to extract 



50 

a person from the photographed image- On the other hand, 
if a type of feeling is determined from an expression or a 
gesture of a person or from voice data and an expression or 
a gesture of a person, it is necessary to extract a person 
from the image . 

In addition, although it is unnecessary to extract a 
person from the image for determination of a feeling, the 
person is preferably extracted when composing an image 
processing pattern, since the image processing pattern is 
preferably composed with respect to the person or his or 
her face. However, if the image processing pattern is 
composed in a specified position in the photographed image, 
or if a background is changed or a density or a color of 
the entire screen is changed, or if falling leaves are 
flown without regard to a position, it is unnecessary to 
extract a person specifically. In this way, extraction of 
a person out of an image may be performed if necessary* In 
addition, it is therefore preferable to make it possible to 
select and set a determining method or the like of a type 
of feeling in advance* Lastly, in step 240, the output 
image is outputted from the image recording apparatus 16. 

Further, in the example described above, the lab side, 
which has received a request of a customer who performed 
photographing by a digital camera or the like, performs 
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image processing by a printer in the lab and prepares a 
print on which a mark emphasizing a feeling is composed. 
However, the above-mentioned image processing may be 
performed on the digital camera side. If the processing is 
performed on the digital camera side, image processing 
patterns or the like are set on the camera side in advance. 

In addition, the above-mentioned processing may be 
entirely performed automatically based on voice data 
recognition and extraction of a face from the photographed 
image, or an operator may input by operation keys or the 
like and perform the processing in accordance with an 
instruction of a customer. 

Also when the processing is performed on the digital 
camera side, image composition processing may be performed 
automatically by incorporating software for executing the 
above-mentioned image processing in the digital camera, or 
a customer may instruct a camera by keys/buttons or the 
like to perform the composition processing at the time of 
photographing . 

In addition, image processing by the above-mentioned 
image processing patterns may be applied to an index print 
while a main print is normally processed . 

In addition, although an image (mark) emphasizing a 
feeling is composed in this embodiment, an expression may 
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be changed by image modification processing including 
morphing processing so as to correspond to a type of 
feeling. For example/ in a case of "anger", the image 
modification processing including the morphing processing 
can be performed to slant eyes upwardly- However, in order 
to enhance amusement aspect of a photograph, a somewhat 
comical expression as in the example shown in Fig. 6 has 
more fun and is effective than a change to a too real 
expression . 

Further, instead of composition corresponding to a 
person in a photographed image as described above r the 
person per se or portions constituting the person such as 
face and body per se may be entirely subjected to 
processing for substituting with an animation image or a 
computer graphics (CG) image, and in particular an 
animation image or a CG image corresponding to a type of 
feeling. 

The animation image or CG image used may be one image 
determined in accordance with the type of feeling, but a 
plurality of patterns can be preferably selected for each 
of a still image and an animation image. 

The animation image and CG image as described above 
may include a content prepared and registered in advance by 
a customer, or a content supplied by a company managing a 
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repeater station as described later. 

As in the above case, conversion patterns can be 
preferably set based on at least one kind of information 
selected from an expression and a gesture of a subject 
person in a photographed image, and voice contents. It 
goes without saying that a request from a customer can be 
more preferably registered. 

The processing as described above enables remarkable 
enhancement of amusement aspect in image expression in a 
photograph, a video, a TV telephone and the like. Further, 
this method is convenient and does not offend an opposite 
party when one does not want to display his or her face in 
a TV telephone or the like. 

A second embodiment of the second aspect of the 
present invention will now be described. 

The second embodiment is an application of the image 
processing method of the second aspect of the present 
invention to an animation such as that in a cellular TV 
telephone or a movie editing rather than to a still image 
such as a photograph . 

In a case of an image displayed on a display screen 
of a cellular TV telephone or an animation such as a 
displayed image of a movie, an image processing method 
itself is basically the same as that in the above-mentioned 
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That is, the same image processing patterns as above 
are registered and image processing software is also 
incorporated in a microcomputer such as a cellular TV 
telephone terminal or the like in advance. Then, in a case 
of a cellular TV telephone, if a registered keyword is 
detected in conversation during a call, processing for 
composing a composition image of a mode corresponding to 
the keyword is performed in the terminal* Then, the 
composition image is transmitted to a terminal of the other 
party of the call and displayed. 

In this case, it is more effective if the displayed 
image itself is also an animation, and a composed "sweat 
mark", for example, as shown in Fig* 6 does not be at a 
standstill but, for example, flows downward little by 
little. 

Further, by securing a sufficient operation speed of 
a CPU, it also becomes possible to perform image processing 
on a real time basis while reproducing an image during a 
call by a cellular TV telephone or during a movie edition. 

A third embodiment of the second aspect of the 
present invention will now be described. 

This embodiment also relates to an apparatus for 
performing image processing on a real time basis such as a 
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TV telephone {or a cellular TV telephone) . That is, this 
embodiment is to determine a type of feeling using 
expression recognition or gesture recognition instead of 
voice data as accompanying information for determining a 
type of feeling of a person. 

For expression recognition,, a table associating 
images of expressions of emotions such as pleasure, anger, 
sorrow and joy with modes corresponding to the expressions 
is registered for each individual in a cellular TV phone 
terminal of each person in advance. In addition, for 
gesture recognition, a table associating gestures 
determined by each person with modes, in such ways as 
raising one finger means this mode and raising two fingers 
means that mode, is registered in a cellular TV telephone 
terminal of each person in advance. 

Then, during a call, a face is extracted from a 
photographed image of a caller and a type of expression is 
identified by the pattern matching in the cellular TV 
telephone terminal. If an expression that coincides with a 
specific expression registered in advance is detected, 
image processing of an image processing pattern of a mode 
corresponding to the registered expression is performed, 
and a processed image is sent to a terminal of the other 
party of the call and displayed. Alternatively, similar 



processing is performed if it is detected that the caller 
gestures in a specific way by the gesture recognition. 

Further, as examples of a publicly known technology 
concerning expression recognition and gesture recognition, 
there are, for example, the technical research report dated 
November 18 and 19, 1999 of the Institute of Electronics, 
Information and Communication Engineers, PRMU99-106 
"Gesture Recognition robust to Variations of a Position of 
Operation" Taiko Amada, Motoyuki Suzuki, Hideaki Goto, 
Shozo Makino (Tohoku University), PRMU99-138 "Automatic 
Extraction of Face Organ Outline and Automation of 
Expression Recognition" Hiroshi Kobayashi, Hisanori 
Takahasi, Kosei Kikuchi (Science University of Tokyo), 
PRMU99-139 "Estimation of Face Region and Expression 
Recognition using Potential Net" Hiroaki Bessho {Image 
Information Science Center), Yoshio Iwai, Masahiko 
Taniuchida (Osaka University), PRMJ99-14Q (Special Lecture) 
"Research on Face Expression Recognition and Image 
Processing Technology" Hiroshi Yamada (Nippon 
University/AIR) , PRMU99-142 (Special Invitation Thesis) 
"Analysis and Recognition of Personal Movement for 
Interaction" Masahiko Taniuchida, Yoshio Iwai (Osaka 
University) , which are preferably applicable to the present 
invention . 
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A fourth embodiment of the second aspect of the 
present invention will now be described- 

This embodiment is to perform image composition 
processing on a repeater station side of the TV telephone, 
although it also relates to a TV telephone or a cellular TV 
telephone. A cellular TV telephone system in this 
embodiment is schematically shown in Fig. 7. Face image 
data (an expression of each mode for determining a mode 
corresponding to a type of feeling) and voice data (a 
keyword for drawing each mode) for each user as well as 
image composition patterns in each mode are registered in a 
database 84 of a repeater station 80 of a communication 
provider through cellular TV telephone terminals €0 and 70* 

During communication, a photographed image of A 
photographed by an image sensor 62 and voices of A and a 
photographed image of B photographed by an image sensor 72 
and voices of B are transmitted to the repeater station 8 0 
from the A' s cellular TV telephone terminal 60 and the B' s 
cellular TV telephone terminal 70, respectively, as shown 
in Fig. 7, 

In a processing section 32 of the repeater station 80, 
faces are always extracted from the photographed images of 
A and B f which are matched with the registered expressions, 
and at the same time, it is checked if registered keywords 
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are heard in the voices in the call* 

Then, for example, if the fret mode is detected in 
the conversation of A, processing for composing, for 
example, a sweat mark is applied to the photographed image 
of A, and the processed image is transmitted to the B' s 
terminal 70. In addition, at this point, a reduced-size 
image of the processed image of A may be transmitted to the 
A' s terminal 60 for confirmation. The face image of B is 
displayed on a display screen 64 of the A' s terminal 60, 
while the processed image of A is displayed on a display 
frame 66 that is provided at the corner of the display 
screen 64 for confirmation of the processed image of A, At 
this point, the processed image of A is similarly displayed 
on a display screen 74 of the B's terminal 70 and, at the 
same time r a reduced^size processed image of B transmitted 
from the repeater station 80 is displayed on a display 
frame 7 6 for confirmation at the corner of the display 
screen 74* 

Further, face image data, voice data, image 
composition patterns and the like of each user may be 
registered in a terminal of a user to perform the process 
from the mode detection to the image composition processing 
on the cellular TV telephone terminal side. In addition, 
the above-mentioned data may be registered in both the 
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repeater station and the terminal such that the processing 
can be performed in either of them. 

In addition, the image composition processing may be 
applied not only to voice and image data to be transmitted 
but also to received voice and image data. For example, in 
Fig. 7, the composition processing with respect to an image 
received from B may be performed in the A' s terminal. In 
this case, amusement aspect is enhanced by adding a pattern 
that the receiver A desires- 

A fifth embodiment of the second aspect of the 
present invention will now be described* 

This embodiment is to correct a position of a 
composition pattern if a misregistration occurs when the 
image composition processing is performed on a real time 
basis . 

Since most of misregistrations of a composition 
pattern are due to a failure in extracting a face, if a 
misregistration of a composition pattern is found on a 
display screen, a face in the photographed image is 
designated by an electronic pen or the like. At this point, 
the outline of the face may be circled using the electronic 
pen, or the eyes may be connected by a line. Moreover, a 
position of the mouth or the like may be designated. 
Alternatively, a position to which the composition pattern 



60 

should be shifted in parallel, an adjustment amount of a 
size or the like may be designated by key operation . 

Thereafter, baaed on the corrected position and size 
of the composition pattern, a face position candidate area 
is automatically corrected to coincide with an original 
composition pattern, and the face extraction processing is 
executed again. Thus, a position and a size of a 
composition pattern is corrected in accordance with a 
movement of a face on a real time basis even in a case of 
an animation as in a cellular TV telephone. Therefore, 
since a composition pattern automatically follows a face 
and is displayed at a predetermined position in a displayed 
image, amusement aspect of an image display in an image 
displaying medium is enhanced. 

In addition, the image processing method according to 
the third aspect of the present invention applies the image 
processing method according to the second aspect of the 
present invention as described above to a TV image captured 
in a personal computer. Specifically, the TV image is 
captured in the personal computer and subjected to 
composition processing, substitution processing, image 
modification processing and color /density modification 
processing as described above. Alternatively, a digital TV 
may have a receiver including the above-mentioned 
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composition processing function and other functions so that 

a customer can further set composition patterns. Amusement 

aspect can be thus enhanced still more. 

The image processing methods according to the second 

and third aspects of the present invention are basically 

configured as described above. 

An image processing method in accordance with a 

fourth aspect of the present invention will now be 

described with reference to Figs. 8 and 9. 

The image processing method in accordance with a 
fourth aspect of the present invention is basically to 
register in advance an area image in a specific area of a 
photographed image or an image characteristic amount and to 
compose on a corresponding area of the photographed image 
or adjust a density and a color tone by using the area 
image or image characteristic amount registered in advance. 

For example, in the image processing method of this 
aspect, reference face images are registered in advance as 
face images of a subject person in a photographed image and 
include an image of a preferred made-up face of the person, 
an image of a favorite face of the person and image 
characteristic amounts thereof, and a face image in a full- 
faced state in which a photographing direction is 
coincident with a line of sight. When the person is a 
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subject of the photographed image, the face of the person 
in the photographed image is corrected, composed or 
converted to the reference face images registered in 
advance including the image of the preferred made-up face 
of the person, the image of the favorite face of the person, 
and the face image in a full-faced state in which the 
photographing direction is coincident with the line of 
sight. Alternatively, the image characteristic amounts 
registered in advance are employed to correct or adjust 
density or color tone of the face so that the finished face 
image has the preferred made-up face or favorite face of 
the person. Thus, anyone can easily correct the face image 
to the reference face images such as the image of the 
preferred made-up face, the image of the favorite face and 
the like, and unnatural feeling due to noncoincidence of 
the line of sight can be removed, 

A case in which a face image of a subject person in a 
photographed image is corrected to an image having a 
preferred made-up face or favorite face or a face image in 
a full-faced state in which a photographing direction is 
coincident with a line of sight will be described below as 
a typical example. However, this is not the sole case of 
this aspect. 

Further, as in the second and third aspects of the 
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p resen t invention, objects to which the fourth aspect of 
the present invention is applied range widely from a 
photograph (still image) to real time image display such as 
a video (animation) and a TV telephone, or the like. 

A first embodiment of the fourth aspect of the 
present invention will be described first. 

The first embodiment is for applying predetermined 
image processing to an image photographed with a digital 
camera or the like by using registration patterns 
registered in advance. 

Fig. 8 is a block diagram schematically showing a 
digital photoprinter including an image processing 
apparatus that implements the image processing method in 
accordance with the first embodiment of the fourth aspect 
of the present invention. A digital photoprinter 90 shown 
in Fig. 8 mainly includes a photographed image acquiring 
device 92 , an image processing apparatus 94, a database 96 
and an image recording apparatus 16. Basically, the image 
recording apparatus 16, the operation system IS and the 
monitor 20 similar to those in the photoprinter 50 shown in 
Fig, 4 can be used* 

The photographed image acquiring device 92 is to read 
photographing image data from a recording medium in which 
the image data are recorded by an image photographing 
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device such as a digital camera. The image processing 
apparatus 94 implements the image processing method in 
accordance with this aspect of the present invention and 
other various kinds of image processing. More specifically, 
the image processing apparatus 94 acquires, extracts or 
prepares reference face images to be registered in advance 
in the database 96 from the photographing image data 
captured by the photographed image acquiring device 92; 
selecrs any of the reference face images registered in 
advance in the database 96; and corrects,- composes or 
converts the face image in the photographed image by using 
the selected reference face image* The database 96 is used 
to register in advance the reference face images including 
an image of a preferred made-up face of the subject person 
in the photographed image, an image of a favorite face of 
the person and image characteristic amounts thereof, and a 
face image in a full-faced state in which a photographing 
direction is coincident with a line of sight {hereinafter 
also referred to as "face image in which line of sight is 
toward camera". Other functions of the image processing 
apparatus 94 and functions of the operation system 18 
including the keyboard 18a and the mouse 18b, the monitor 
20, and the image recording apparatus 16 are basically the 
same as those in the digital photoprinter 50 shown in Fig. 
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4. Therefore, the description will be omitted. 

Image processing of the first embodiment will be 
hereinafter described with reference to a flow chart of Fig* 
9. 

First, in step 300, a face is photographed in advance 
in the most preferable situation by a photographing device 
such as a digital camera and the photographed image of the 
face is acquired as a photographed image for registration 
in the photographed image acquiring device 92. 

Then, in step 310, a face extraction algorithm is 
applied in the image processing apparatus 94 to extract a 
face region for a reference face image from the 
photographed face image acquired by the photographed image 
acquiring device 92 . Image characteristic amounts of the 
reference face image may be calculated in this process. It 
should be noted that calculation of the image 
characteristic amounts of the reference face image and 
hence the extracting process in step 310 may be omitted 
when the whole of the photographed face image is used as 
the reference face image. When omitted in step 310, 
calculation of the image characteristic amounts of the 
reference face image may be performed in the preceding step 
300, subsequent step 320 or step 360 for correction process 
to be described later* 
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In step 320, the whole of the photographed face image 
or the face region extracted therefrom is registered in the 
database 96 as the reference face image. Pre-processing is 
thus finished. 

For the reference face images to be registered in the 
database 96, any type of faces including a neatly made-up 
face, a face with hair set in a beauty salon or barbershop, 
a shaven face and a face with mustache set may be used. A 
plurality of reference face images prepared for a specific 
person may be changed in use for example between morning 
and night/ for each season or for each party in the case of 
displaying in a TV telephone or a cellular telephone. 

Each portion constituting the face image may be 
registered as the reference pattern in the reference face 
image. Namely, in the extracting process in step 310, an 
area which it is desired to use for correction or 
composition may be automatically extracted or manually 
designated to register the thus extracted or designated 
area as the reference pattern. Examples of the reference 
pattern include respective portions constituting the face 
such as eyes r eyebrows, cheeks, lips, nose, ears, hair,, and 
mustache, and accessories such as earrings, a hat {bandanna 
and headband also included} and glasses (sunglasses also 
included) . 
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In this case, it is more preferable that the face 
extraction algorithm can be applied to automatically 
recognize which portion of the face the designated area 
indicates. When the accuracy in face extraction is low, 
area setting is performed while designating the respective 
portions one by one. 

Then, in step 330, the photographed image is acquired 
by the photographed image acquiring device 92. The process 
of acquiring the photographed image in step 330 precedes 
for example transmission or display of the photographed 
image to or on the monitor screen of a TV telephone or a 
cellular telephone. Thus, in the communication with such a 
telephone, an owner or an exclusive user of the TV 
telephone or the cellular telephone is very often a subject 
person in the photographed image. The subject person is 
specified in many cases* Therefore, it is preferable that 
information such as ID can be automatically acquired. 
Further, when there are many users, each user is given an 
ID so that the ID of a subject person can be acquired in 
step 330 together with the photographed image by the 
photographed image acquiring device 92. 

Then, in step 340, the image processing apparatus 94 
applies the face extraction algorithm to the photographed 
image acquired by the photographed image acquiring device 



68 

92 to extract the face region of the subject person from 
the photographed image as the face image. In this process , 
it is preferable that not only the entire face region but 
also the respective portions constituting the face image 
are determined. 

For example, the contour of the face is first 
extracted, followed by area restriction of the respective 
portions, after which the areas of the respective portions 
can be determined finally by means of configuration pattern 
matching.. 

If the accuracy in face extraction is low, it is 
preferable that, when photographing is started with a 
photographing device such as a digital camera, the subject 
person takes a posture so that the face of the subject 
person comes to (or is located in) the position within a 
guide such as a face frame displayed on a viewfinder or 
display device* Extraction of the face image by means of 
the configuration pattern matching is thus facilitated, and 
once having been caught, the face of the subject can be 
followed by means of local matching between photographing 
frames even in a different photographing frame. 

In step 350, a registered reference face image of a 
person that the image processing apparatus 94 searched or 
specified from the database 96 based on the above-mentioned 
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ID or the like, a reference pattern and optionally image 
characteristic amounts thereof are selected and called. It 
is also possible to have new combinations by selecting from 
different reference images for the respective portions of 
the face image of the specific person as described above* 
For example, a newly combined face image including a first 
reference face image for eyes, a third reference face image 
for eyebrows and an eighth reference face image for 
earrings can be also made* 

Either of steps 340 and 350 may be preceded or both 
may be performed simultaneously* 

Then, in step 360, the face image extracted in step 
340 and the areas of the respective portions constituting 
the face image are corrected, composed or substituted by 
using the registered reference face image for the specific 
person, reference pattern and their image characteristic 
amounts selected in step 350, The registered reference 
face image may be composed on or substituted for the entire 
face image, or a registered reference pattern image may be 
used for each portion of the face to perform correction, 
composition or substitution. Alternatively, image 
characteristic amounts of registered reference pattern 
images may be used to perform correction* 

For example, cheeks and lips are corrected to have a 
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color tone of the reference pattern of the corresponding 
portions, In particular, lips move and hence, only the 
color tone of lips is preferably corrected after the lip 
area is separated from the skin area in color tone on a 
pixel basis. 

As for hair, eyebrows, cheeks, ears {earlobes}, 
reference patterns are preferably overwritten for 
composition. In this case, a portion surrounding the 
composition area is preferably smoothed to have a blurred 
boundary. 

Finally, in step 370, the photographed image in which 
the face image is corrected in step 360 is outputted from 
the image processing apparatus 94 as an output image. The 
output image may be outputted from the printer 16 as a 
photographic print or displayed on the monitor 20, or 
outputted to a TV telephone or cellular telephone terminal 
(more specifically its image processor; see Fig. 7) so as 
to transmit to a TV telephone or cellular telephone 
terminal (more specifically its image processor; see Pig. 
7) on an opposite party side for displaying on a display 
screen. Alternatively, one may directly transmit the 
photographed image to the opposite party side to display on 
the display screen, or of course display on a display 
screen of his or her apparatus. 



Even inexperienced or unskilled persons in personal 
computer or image processing software can easily correct 
images so as to have a preferred made-up face or a favorite 
face. 

In addition, a clothing portion (area set under the 
face) or a background (region other than the face area) of 
the subject person may be enchased with decoration patterns 
so that intimate apparel and night clothes including 
pajamas or interior of a room can be hidden without giving 
unpleasant feeling to others. 

Then, a second embodiment of the fourth aspect of the 
present invention will be described* 

The second embodiment is for applying predetermined 
image processing to an image photographed with a digital 
camera or the like by using registration patterns 
registered in advance as in the first embodiment, but is 
different from the first embodiment in that the 
registration pattern registered in advance is a face image 
in which a photographing direction is coincident with a 
line of sight of a subject person (face image in which line 
of sight is toward camera) . Therefore, the second 
embodiment will be also described below with reference to 
Figs. 8 and 9. 

According to the second embodiment, in step 300 shown 
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in Fig, 9, a photographed image is taken with a 
photographing device such as a digital camera in a state in 
which a subject looks at the camera,, and then recorded. 
The recorded photographed image is acquired by the 
photographed image acquiring device 92 (see Fig, 8) as a 
photographed image for registration. 

Then, in step 310, the image processing apparatus 94 
applies the face extraction algorithm to the photographed 
image for registration acquired by the photographed image 
acquiring device 92 to extract an area of eyes as a 
reference pattern- In step 320, the thus extracted area of 
eyes is registered in the database 96 as a reference 
pattern. Pre-processing is thus finished. 

Then, in step 330 , the photographed image is acquired 
by the photographed image acquiring device 92 , The process 
of acquiring the photographed image in step 330 precedes 
for example transmission or display of the photographed 
image to or on the monitor screen of a TV telephone or a 
cellular telephone* 

Then, in step 340, in the communication with such a 
telephone, the image processing apparatus 94 applies the 
face extraction algorithm to the photographed image 
acquired by the photographed image acquiring device 92 to 
extract the area of eyes of the subject person from the 
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photographed image. 

In step 350, the image processing apparatus 94 
selects and calls a registered reference pattern from the 
database 96. Either of steps 340 and 350 may be preceded 
or both may be performed simultaneously. 

Then, in step 360, the area of eyes extracted in step 
340 is corrected, composed or substituted by using the 
registered reference pattern selected in step 350. When 
movement between image frames is small or when the image 
can be considered to be in a full-faced state (as a result 
of balance check between left side and right side) , the 
registered reference pattern image may be used as such for 
composition . 

Finally, in step 370, a photographed image in which 
eyes have been corrected in step 360 so as to look at the 
camera is outputted from the image processing apparatus 94 
as an output image. Alternatively, in a TV telephone or a 
cellular telephone terminal, a stationary state of the 
photographed image (for example a state of an image 
photographed on startup in which a subject is correctly 
postured and looking at a monitor) is temporarily stored. 
When the difference between the temporarily stored image 
and the actual image exceeds a specified value, the 
photographed image can be transmitted as it is in order to 
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prevent unnatural feeling due to the large movement. 

The output image may be outputted from the printer 16 
as a photographic print or displayed on the monitor 20, or 
outputted to a TV telephone or cellular telephone terminal 
so as to transmit to a TV telephone or cellular telephone 
terminal on an opposite party side for displaying on a 
display screen. Alternatively, the photographed image may 
be directly transmitted to the opposite party side to 
display on the display screen. 

Unnatural feeling due to noncoincidence of the line 
of sight can be thus removed. 

In addition, when a reference pattern is registered, 
an animation of a photographed image in which a subject 
looks at a camera, for example a photographed animation in 
which a subject winks at least once is recorded, As for 
composition during the communication with a TV telephone or 
the like,, composition can be made as an animation 
irrespective of the actual timing of winking. 
Alternatively, when a photographed image is acquired, 
winking detection may be performed for synchronization, 
which is more preferable. 

In addition, in a third embodiment of the fourth 
aspect of the present invention, if a transmission terminal 
such as a TV telephone or cellular telephone terminal (see 



75 

Fig. 7) has a high arithmetic capability, various types of 
correction processing as described in the first and second 
embodiments of this aspect can be executed in the 
transmission terminal. Alternatively, a reference face 
image and decoration pattern data may be registered in the 
repeater station (see Fig. 7) in combination with a 
customer ID and executed in the repeater station. 

When family members use a TV telephone, a cellular 
telephone terminal or the like in the aspect under 
consideration, it is preferable to register a reference 
face image and a reference pattern for each family member 
as in the first aspect as described above . 

The image processing method of the fourth aspect of 
the present invention is basically configured as described 
above * 

As described above in detail , according to the first 
aspect of the present invention, an output image on which 
individual preference of each customer is reflected can be 
automatically obtained and amusement aspect of photography 
can be enhanced. 

In addition, as described above in detail, according 
to the second aspect of the present invention, since a 
content that is desired to be emphasized according to a 
feeling is automatically visualized and represented in the 
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display of, in particular, a face of a person in an image, 
amusement aspect in image representation such as a 
photograph, a video, a TV telephone or the like can be 
significantly enhanced. In addition, the automatic 
visualization is convenient and does not offend an opposite 
party when one does not want to display his or her face in 
a TV telephone* 

As also described above in detail, the present 
invention overcomes the above-mentioned prior art problems 
so that even inexperienced or unskilled persons in personal 
computer or image processing software can easily correct 
images so as to have a preferred made-up face or a favorite 
face and remove unnatural feeling due to noncoincedence of 
the line of sight. 

Thus, it is seen that an image processing method is 
provided* One skilled in the art will appreciate that the 
present invention can be practiced by other than the 
preferred embodiments which are presented for the purposes 
of illustration and not of limitation, and the present 
invention is limited only by the claims which follow. 



